system with a random number generator used to remove contouring in cmos imager data having an extended dynamic range

ABSTRACT

The present invention is a method of image data processing that includes determining whether the image data indicates photodiode saturation, and removing contouring from the image data caused by the saturation. The contour artifact removal is accomplished by adding a random number to adjusted photodiode data from an extended range imager when the data from the imager indicates that the photodiode has been saturated. The adjusted photodiode data is a sum of photodiode data and spillover rate adjusted diffusion data.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. application Ser. No. 10/387,710filed Mar. 13, 2003, now allowed.

FIELD OF THE INVENTION

The present invention relates generally to the field of imaging, and inparticular to imaging using an extended dynamic range imager. Morespecifically, the invention relates to removing a contour from the imagedata caused when a photodiode of the imager is saturated and chargespills into a floating diffusion region where recovering the full pixelvalue caused the contour.

BACKGROUND OF THE INVENTION

Imagers, particularly CMOS imagers, in the past have had problems withscenes that have a high dynamic range such that parts of the scene havehighlights therein. In these portions of the image, the pixels of thesensor become saturated such that the digital pixel outputs are all 1's.To solve this problem, sensors have been designed for enhanced dynamicrange. (See, for example, the extended dynamic range imagers discussedin European Patent Office Applications EP 1096789 A2 and EP 1096790 A2and incorporated by reference herein). These sensors have a photodetector and a floating diffusion into which excess charge (that abovephotodiode saturation) is spilled. These extended range sensors outputtwo values for each pixel, the photodiode value and a spillover valuethat indicates the rate of charge spillover from the saturatedphotodiode to the floating diffusion. These two different values areused to create the viewable image of the captured scene; when there isno spillover, the photodiode value is used; and when there is spillover,a new pixel value is created based on the sum of the photodiode valueand the floating diffusion value multiplied by the ratio of thephotodiode integration time to the floating diffusion integration time.As the ratio of the photodiode integration time to the floatingdiffusion integration time is increased, the magnitude of the multiplierincreases. This multiplication causes a contouring artifact to appear inthe resultant image. This is a drawback of the current state of the art.

For example, take the case where a photodiode of a conventional imagerhas a dynamic range of 8 photographic stops. This can be represented ina linear, binary system, with 8 bits of dynamic range. This correspondsto a system dynamic range of 48 db, for the photodiode. Also assume thatthe floating diffusion has a charge capacity (when transformed from rateof spillover information to pixel information), that represents anadditional 5 stops of photographic range, and then this corresponds toan additional 5 upper significant bits of pixel information, 29 dbadditional dynamic range. The overall dynamic range of this system wouldbe 13 bits, 13 photographic stops, and 77 db. To capture a picture withthis imager, it is necessary to operate the imager with 200 lines ofintegration time for the photodiode and 25 lines of integration time forthe floating diffusion. This means that to convert from floatingdiffusion rate of spillover data to pixel data the floating diffusionrate of spillover data must be multiplied by 8 (200 Lines Pd/25 LinesFd) and then added to the photodiode data for that pixel, see equation 6in application EP 1096789 A2.

A typical Extended Dynamic Range (EDR) CMOS Imager with one output willoutput serial pixel digital data in a single channel 10. This imagerprovides two values per pixel as depicted in FIG. 1. The first value(Fd) is the signal representing the rate of charge spillover from thesaturated photodiode to the pixel floating diffusion (if the light levelis high enough to cause saturation of the photodiode). The second value(Pd) of the same pixel is the value on the pixel photodiode. When thephotodiode is not saturated, the image information is the pixel value onthe photodiode. The residual floating diffusion information in such asituation is the floating diffusion dark current and any signalresulting from of undesirable partial light sensitivity of the floatingdiffusion; this signal on the floating diffusion is also present whenthe photodiode is saturated.

Conventionally, a transform is used to convert from floating diffusionrate of spillover information and photodiode value, to pixel finalnumerical value (see, for example, equation 6 in application EP 1096789A2). A new pixel final numerical value is created based on the sum ofthe photodiode value and the floating diffusion value multiplied by theratio of the photodiode integration time to the floating diffusionintegration time. The multiplication is done to convert from rate offloating diffusion spillover information to actual pixel value. When thephotodiode is not bloomed (i.e. not saturated), the floating diffusiondoes not contain image information, and the pixel data is the photodiodedata.

Typically, the pixel values are analog-to-digital (A/D) converted andthe transform is implemented in digital logic or a digital signalprocessor. The multiplication by the ratio of the photodiode integrationtime to the floating diffusion integration time results in an image thathas contouring in the high dynamic range areas of the image. The picturepixel information exists only in the sum of transformed floatingdiffusion information and the photodiode information What is needed is asystem that will remove the contouring caused by multiplying thefloating diffusion information by the ratio of the photodiodeintegration time to the floating diffusion integration time.

SUMMARY OF THE INVENTION

The present invention is directed to overcoming one or more of theproblems set forth above. Briefly summarized, according to one aspect ofthe present invention, a method of image data processing includesdetermining whether the image data indicates photodiode saturation, andremoving contouring from the image data caused by multiplying thefloating diffusion information by the ratio of the photodiodeintegration time to the floating diffusion integration time. The contourremoval is accomplished by adding a random number to the result of thefloating diffusion value multiplied by the ratio of the photodiodeintegration time to the floating diffusion integration time when datafrom the imager indicates that the photodiode has been saturated.

The present invention utilizes a random number generator to remove thecontouring caused when operating an active pixel CMOS imager in extendeddynamic range mode. Typically the LSB values that signify the contentsof the saturated photodiode are set to “1” because of the photodiodesaturated condition. A contouring artifact is caused to appear in theresultant image due to the method used to combine the photodiodeinformation with the floating diffusion information to form a finalpixel value. The present invention removes the contouring artifact byadding a random number to the final pixel value. The number of bits ofprecision of the random number is determined by the multiplicationfactor used to convert the floating diffusion rate of spillover data tothe final pixel data. In the above-discussed example, the multiplicationfactor is 8. (Multiplying by 8 causes the 3 least Significant bits ofthe result to be 0. This is the contour in the pixel data). Thecorresponding random number will have 4 (2̂3) bits of precision. Therandom number generator and associated logic can be integrated on thesame silicon substrate as the CMOS imager.

The above and other objects of the present invention will become moreapparent when taken in conjunction with the following description anddrawings wherein identical reference numerals have been used, wherepossible, to designate identical elements that are common to thefigures.

These and other aspects, objects, features and advantages of the presentinvention will be more clearly understood and appreciated from a reviewof the following detailed description of the preferred embodiments andappended claims, and by reference to the accompanying drawings.

Advantageous Effect of the Invention

The present invention has the advantages discussed below.

The current state of the art suffers from a drawback. The current stateof the art provides two values per pixel. The first value is the signalrepresenting light integrated on the pixel photodiode. The second valueis the value on the floating diffusion. When the photodiode is bloomed,the floating diffusion value represents the RATE of charge spilloverfrom the photodiode to the floating diffusion. A transform is used toconvert from rate of spillover information, and photodiode value, topixel final numerical value. Since the photodiode is bloomed (saturated)the numerical value of the photodiode portion of the final pixel valueis at maximum (At or close to all 1's). Typically the pixel values areanalog to digital converted and the transform is implemented in digitallogic or a digital signal processor. The conversion of the floatingdiffusion rate of charge spillover value to final pixel value results inan image that has contouring. The picture information exists only in thetransformed floating diffusion information, and in the fact that thephotodiode is saturated, indicating that the floating diffusiontransformed value plus the photodiode saturated value is to be used asthe final pixel value in image processing. The present inventionadvantageously utilizes a random number generator to remove thecontouring caused by the floating diffusion transformed value portion.

The present invention advantageously improves image quality of imagescaptured by extended dynamic ranger imagers.

The present invention also advantageously provides image data thatproduces a more pleasing image having an extended dynamic range.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts the format of pixel data produced by an extended dynamicrange imager.

FIG. 2 shows the conversion of the imager pixel data into parallelstreams of diffusion data (Fd) and photodiode data (Pd).

FIG. 3 is a block diagram of the circuits of a hardware embodiment ofthe present invention.

FIG. 4 illustrates the separator circuit 30 of FIG. 3 in more detail.

FIG. 5 is a timing diagram for the circuit 30 of FIG. 4.

FIG. 6 illustrates the contour removal and reconstruction circuit 32 ofFIG. 3 in more detail.

FIG. 7 depicts the format of output image data.

FIG. 8 shows how diffusion and photodiode data are combined.

FIG. 9 depicts the conversion or transformation of pixel data whenphotodiode saturation occurs.

FIG. 10 depicts the pixel data when photodiode saturation does notoccur.

FIG. 11 is a flowchart of a software embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, the present invention will be described inthe preferred embodiment as hardware. Those skilled in the art willreadily recognize that the equivalent of such hardware may also beconstructed in software.

The present invention utilizes a random number generator to remove thecontouring caused when operating the conventional extended dynamic rangeactive pixel CMOS imager and the photodiode/photodetector becomessaturated. The present invention removes the contouring artifact byadding a random number to the final pixel value formed by adding thefloating diffusion rate of spillover value, multiplied by the ratio ofthe photodiode integration time to the floating diffusion integrationtime, to the photodiode value. A random number generator and associatedlogic can be integrated on the same silicon substrate as the CMOSimager. Alternately, this invention may be executed utilizing softwareand a microprocessor (see FIG. 11).

To facilitate the addition of a random number to the pixel value inhardware, the data is first converted from serial data to parallel dataas depicted in FIG. 2 where a first channel 20 includes the diffusiondata (Fd) and a second channel 22 includes corresponding photodiode data(Pd). A serial data separator circuit 30, as depicted in FIG. 3,accomplishes this conversion. The converted data then has the contourartifact removed and the transformed data is then reconstructed intoserial image data as in FIG. 1 by a contour removal and reconstructioncircuit 32 also shown in FIG. 3. The contour corrected pixel data isthen further processed to create an image, which involves the storage ofthe pixel data in an image memory.

Depicted in more detail in FIG. 4 are the components of the separatorcircuit 30. The separator 30 operates to arrange, parallel in time, thatthe pixel floating diffusion data and the pixel photodiode data existsimultaneously for the respective final pixel value. FIG. 5 depicts theclock and data signals of the components of FIG. 4. The serial pixeldata (Fd, Pd) arrives from the imager 40 responsive to a pixel clocksignal ck1 supplied to the imager 40 by a clock signal source 42 throughan inverter 44. The diffusion data (Fd) is clocked through storageregisters 46 and 48 responsive to the pixel clock signal and asubdivided clock signal ck2 supplied by divider 50. The pixel data (Pd)is clocked through register 52 by clock signal ck3.

The parallel pixel data (Pd) and diffusion data (Fd) is supplied to thecontour artifact removal and reconstruction circuit 32 shown in moredetail in FIG. 6. This circuit 32 adds a random number generated by arandom number generator 66 to the final pixel value, if the magnitude ofthe associated photodiode value is at, or close to, saturation. Therandom number can be generated in a number of different ways such asthat described in U.S. Pat. No. 5,101,452, incorporated by referenceherein. The diffusion data Fd is supplied to a two-port multiplier 62.The second input to the multiplier 62 is the numerical representation ofthe ratio of the photodiode integration time to the floating diffusionintegration time. In a camera, this information would typically comefrom the microprocessor, or digital signal processor, that executes theimage exposure algorithm. The multiplier 62 has a third input, theenable. When the enable is a 1 the multiplier 62 outputs the product ofthe floating diffusion spillover data and the integration time ratio.When the enable is a 0 the multiplier 62 outputs a 0. The multiplier 62enable is connected to the output of a magnitude comparator 68. Themagnitude comparator 68 is used to establish the photodiode saturationthreshold. The variable Z is a constant that can be all 1's or a smallernumber and is used to accommodate the situation where the photodiode hasa soft saturation. The floating diffusion spillover may start before thephotodiode is completely saturated. The multiplier 62 output isconnected to the A input port of a three port adder 64. The B input portof the three-port adder 64 is connected to the output of the bit selectlogic or selector 66. The bit select logic performs two functions.First, it selects how many bits of the random number generator 66 to addto the sum of the photodiode pixel value and the product of theintegration time ratio and the floating diffusion spillover pixel data.This is accomplished with the select inputs to the bit select logic 66.In the previously discussed example, the gain ratio was 8. In thissituation 8 is 2̂3. Four (4) bits of random noise (starting from theleast significant bit) are required to remove the contour caused bymultiplying the floating diffusion spillover data by 8. The secondfunction of the bit select logic is to gate the random number source offwhen the photodiode is not saturated. This is accomplished with theenable input of the bit select logic 66. This data reconstructioncircuit 32 preferably implements the method of equation 6 of EP 1096789previously mentioned either in hardware as discussed above or throughsoftware, such as a look-up table. In this equation, when bloomingoccurs, the reconstructed pixel value is the saturated photodiode signalplus the product of the floating diffusion spill over signal and theratio of the photodiode integration time to the floating diffusionintegration time. When blooming does not occur, the photodiode data isthe pixel data.

Take the case where the scene has very bright highlights that need to becaptured as part of the picture. Typically, an appropriate photodiodeintegration time is used to capture the scene, not including thehighlights. The floating diffusion integration time is chosen such thatthe floating diffusion signal is close to, but not, saturated, in thehighlight areas of the scene. Since the light in the highlight areas ishigh, the spillover rate onto the floating diffusion is high. Theintegration time of the floating diffusion will be short. An examplemight be 200 lines of integration time for the photodiode and 25 linesof integration time for the floating diffusion. According to equation 6of EP 1096789 we multiply the floating diffusion value by 200/25 and addit to the photodiode data to obtain the final pixel value. This finalpixel value is quantization precision limited by the ratio of thephotodiode integration time to the floating diffusion integration time,the multiplier. For each factor of 2-multiplier increase, the datalooses a least significant bit (LSB) of quantization precision,(binary). This is classic imager contouring. The present invention addsa random number to remove the apparent contouring in the resultantimage. The bit depth of the random number is determined by the ratio ofthe photodiode integration time to the floating diffusion integrationtime. Therefore the new equation becomes:

Veff=Vout

for Vout<Vpdsat

Veff=Vpdsat+(Vout−Vpdsat)(Tpdint/Tfdint)+RandomValue

for Vout>=Vpdsat  (1)

where Veff is the final pixel data, Vout is the total voltage availablefrom the pixel, (if it had not saturated), Vpdsat is the photodiodesaturation voltage, Tpdint is the integration time of the photodiode andTfdint is the integration time of the floating diffusion region.

The magnitude comparator 68 can alternately be connected to the floatingdiffusion data. The floating diffusion information could contain darkcurrent and/or the floating diffusion may be slightly light sensitive.This light sensitivity is an undesirable behavior. The magnitudecomparator can cause equation 1 data reconstruction if the floatingdiffusion information is greater than some minimum value. This minimumvalue represents the dark current and the undesired light sensitivitysignal and can be called a non-spillover threshold.

FIG. 7 graphically illustrates the image data 70 that is the output fromthe circuit of FIG. 6. This data has one value per pixel and has apossible numerical range greater than that of the photodiodeinformation. In this illustration, the floating diffusion spillovervalue has 8 bits precision. The photodiode data has 8 bits in precision.The resultant data will have precision of 8 bits+8 bits+N bits, where Nbits represents the number of bits used to express the ratio of thephotodiode integration time to the floating diffusion integration time.The combination of the floating diffusion data (Fd) 72 and the pixeldata (Pd) 74, through the use of equation 1, to produce a pixel 76 isdepicted graphically in FIG. 8.

FIG. 9 shows the binary transformation of a typical pixel value that hasan amplitude that is high enough such that the floating diffusion data(Fd) is transformed and a random number is added to the data. Thisexample represents 8 bits of photodiode data and 8 bits of floatingdiffusion data. The upper pixel value 100 is the data from the imagerand the lower pixel value 102 is the data after randomization. This datastructure used to control the production of an image by a computerincludes the floating diffusion data, the saturated photodiode data, andthe random number when saturation of the photodiode has occurred.

FIG. 10 shows the binary transformation (or lack thereof) of a typicalpixel value that has an amplitude that is not high enough for thefloating diffusion data (Fd) to be transformed and a random number to besubstituted for the photodiode data. The photodiode data (Pd) becomesthe raw pixel data. This example also represents 8 bits of photodiodedata and 8 bits of floating diffusion data. The upper pixel value 110 isthe data from the imager and the lower pixel value 112 is the data afterit passes through the circuits of the present invention. In this datastructure the least significant bits are photodiode data. This is anidealized illustration. In practice the floating diffusion informationcontains dark current and possibly a small signal due to undesired lightsensitivity. The present invention as discussed above does not addressthis issue since the magnitude comparator 68 decides when to processfloating diffusion data responsive to or based on photodiode data, notfloating diffusion data. The issue is addressed below.

As previously mentioned, the contour removal of the present inventioncan be accomplished via software using a microprocessor and a flowchartshowing the operations of such software is depicted in FIG. 11. When thephotodiode data is input 120, a determination is made 122 whether thephotodiode data indicates that a spillover into the floating diffusionregion has occurred using a blooming threshold Z compared to thephotodiode data. If so, a calculation 124 is performed that adds thefloating diffusion rate of spillover value, multiplied by the ratio ofthe photodiode integration time to the floating diffusion integrationtime, to the photodiode value. A random number is added 126 to removethe apparent contouring in the resultant image, caused by the multiplierin the prior calculation. The bit depth of the random number isdetermined by the number of bits it takes to express the ratio of thephotodiode integration time to the floating diffusion integration time.The reconstructed pixel data is then output 128. If the photodiode dataindicates that a spillover into the floating diffusion region has notoccurred, then the photodiode data is output 130 as the pixel data.

Note that the operations 124 and 126, and the equivalent operations inthe hardware version, can be accomplished by using a look-up-table (LUT)that has as inputs the floating diffusion data, the photodiode data andthe random number.

In the present invention, if the floating diffusion is not partiallylight sensitive, it is possible to connect the magnitude comparator tothe floating diffusion spill over data source instead of the photodiodedata source. Alternately an OR gate could be used, in place of themagnitude comparator, with the floating diffusion data, skipping a fewleast significant bits (LSB's), to allow for dark current accumulation.

The system also includes permanent or removable storage, such asmagnetic and optical discs, RAM, ROM, etc. on which the process and datastructures of the present invention can be stored and distributed. Theprocesses can also be distributed via, for example, downloading over anetwork such as the Internet.

The invention has been described with reference to a preferredembodiment. However, it will be appreciated that variations andmodifications can be effected by a person of ordinary skill in the artwithout departing from the scope of the invention.

PARTS LIST

-   10 serial pixel data-   20, 72 floating diffusion data-   22, 74 photodiode data-   30 serial data separator circuit-   32 contour removal and reconstruction circuit-   40 imager-   42 clock signal source-   44 inverter-   50 divider-   52 register-   62 multiplier-   64 adder-   66 random number generator-   68 comparator-   70 output image data-   76 output pixel-   100, 110 upper pixel value-   102, 112 lower pixel value-   120-130 software operations

1. A computer readable data structure for controlling production of animage by a computer, comprising pixel data having most significant bitscontrolled by a sum of spillover rate adjusted diffusion data andphotodiode data and least significant bits being a sum of leastsignificant bits of the spillover rate adjusted floating diffusion dataand the photodiode data with a random number when the photodiode dataindicates photodiode saturation.
 2. A computer readable storagecontrolling a computer by determining whether image data indicatesphotodiode saturation and removing contouring from the image data causedby the photodiode saturation.